Generation of Depth Data Based on Spatial Light Pattern

ABSTRACT

Technologies are generally described for generating depth data based on a spatial light pattern. In some examples, a method of generating depth data includes obtaining an image of one or more objects on which a spatial light pattern is projected, wherein blurring of the spatial light pattern in the image monotonously increases or decreases in a depth direction, calculating a value of a spatial frequency component of the image in a local image area around a pixel of interest, and determining depth data corresponding to the calculated value of the spatial frequency component by utilizing a preset relationship between depths and values of the spatial frequency component.

TECHNICAL FIELD

The disclosures herein generally relate to the generation of depth data.

BACKGROUND

Unless otherwise indicated herein, the approaches described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Images for stereoscopic viewing may be composed of an image for theright eye and an image for the left eye. The observer may typically usespecial glasses that make the right eye see only the image for the righteye and the left eye see only the image for the left eye, thereby givingthe observer the perception of a three-dimensional scene. There may betwo typical methods for transmitting image data for stereoscopicviewing. The first method may involve sending both left-eye andright-eye images, which may result in twice as much data as normal2D-image data being transmitted. The second method may involve creatinga depth map from a left-eye image and a right-eye image and thentransmitting the resulting depth map together with either the left-eyeor right-eye image. The recipient of the data may use the depth map torecreate one of the two images that has not been sent. The second methodmay also result in a larger amount of data than normal 2D-image databeing transmitted. Further, the need to generate a depth map at atransmission side may impose additional computation load.

SUMMARY

According to one or more embodiments, a method of generating depth dataincludes obtaining an image of one or more objects on which a spatiallight pattern is projected, wherein blurring of the spatial lightpattern in the image monotonously increases or decreases in a depthdirection, calculating a value of a spatial frequency component of theimage in a local image area around a pixel of interest, and determiningdepth data corresponding to the calculated value of the spatialfrequency component by utilizing a preset relationship between depthsand values of the spatial frequency component.

According to one or more embodiments, an apparatus for generating depthdata includes an image receiving unit configured to receive an image ofone or more objects on which a spatial light pattern is projected,wherein blurring of the spatial light pattern in the image monotonouslyincreases or decreases in a depth direction, a frequency componentcalculating unit configured to calculate a value of a spatial frequencycomponent of the image in a local image area around a pixel of interest,and a depth map generating unit configured to determine depth datacorresponding to the calculated value of the spatial frequency componentby utilizing a preset relationship between depths and values of thespatial frequency component.

According to one or more embodiments, a system for generating depth dataincludes a projection unit configured to project a spatial lightpattern, an imaging unit configured to capture an image of one or moreobjects on which the spatial light pattern is projected, whereinblurring of the spatial light pattern in the image monotonouslyincreases or decreases in a depth direction, a frequency componentcalculating unit configured to calculate a value of a spatial frequencycomponent of the image in a local image area around a pixel of interest,and a depth map generating unit configured to determine depth datacorresponding to the calculated value of the spatial frequency componentby utilizing a preset relationship between depths and values of thespatial frequency component.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the drawings and the followingdetailed description.

BRIEF DESCRIPTION OF DRAWINGS

The subject matter of the present disclosure is particularly pointed outand distinctly claimed in the concluding portion of the specification.The foregoing and other features of the present disclosure will becomemore fully apparent from the following description and appended claims,taken in conjunction with the accompanying drawings. Understanding thatthese drawings depict only several embodiments in accordance with thedisclosure and are, therefore, not to be considered limiting of itsscope, the disclosure will be described with additional specificity anddetail through use of the accompanying drawings.

[FIG. 1]FIG. 1 is a drawing illustrating an example of a depth datagenerating system;

[FIG. 2]FIG. 2 is a drawing illustrating an example of a spatial lightpattern;

[FIG. 3]FIG. 3 is a drawing illustrating an example of the blurring of aspatial light pattern that monotonously increases in the depth directionon an object surface;

[FIG. 4]FIG. 4 is a block diagram illustrating an example of theconfiguration of a depth data generating apparatus;

[FIG. 5]FIG. 5 is a flowchart illustrating a method for generating depthdata performed by the depth data generating apparatus;

[FIG. 6]FIG. 6 is a block diagram illustrating an example of theconfiguration of a reflectance correcting unit;

[FIG. 7]FIG. 7 is a flowchart illustrating the process of performingluminance correction performed by the reflectance correcting unit;

[FIG. 8]FIG. 8 is a block diagram illustrating an example of theconfiguration of a depth map generating unit;

[FIG. 9]FIG. 9 is a flowchart illustrating a method for determiningdepth data performed by the depth map generating unit; and

[FIG. 10]FIG. 10 is a drawing illustrating an example of a monotonousblurring increase in the depth direction; all arranged in accordancewith at least some embodiments described herein.

DESCRIPTION OF EMBODIMENTS

The following detailed description sets forth various examples alongwith specific details to provide a thorough understanding of claimedsubject matter. It will be understood by those skilled in the art,however, that claimed subject matter may be practiced without some ormore of the specific details disclosed herein. Further, in somecircumstances, well-known methods, procedures, systems, componentsand/or circuits have not been described in detail in order to avoidunnecessarily obscuring claimed subject matter. In the followingdetailed description, reference is made to the accompanying drawings,which form a part hereof. In the drawings, similar symbols typicallyidentify similar components, unless context dictates otherwise. Theillustrative embodiments described in the detailed description,drawings, and claims are not meant to be limiting. Other embodiments maybe utilized, and other changes may be made, without departing from thespirit or scope of the subject matter presented here. It will be readilyunderstood that the aspects of the present disclosure, as generallydescribed herein, and illustrated in the Figures, can be arranged,substituted, combined, separated, and designed in a wide variety ofdifferent configurations, all of which are explicitly contemplatedherein.

This disclosure is drawn, inter alia, to methods, apparatus, devices,and/or systems related to the generation of depth data.

Briefly stated, technologies are generally described herein for a methodof and apparatus for generating depth data. In some examples, the methodmay include obtaining an image of one or more objects on which a spatiallight pattern is projected, wherein blurring of the spatial lightpattern in the image monotonously increases or decreases in a depthdirection, calculating a value of a spatial frequency component of theimage in a local image area around a pixel of interest, and determiningdepth data corresponding to the calculated value of the spatialfrequency component by utilizing a preset relationship between depthsand values of the spatial frequency component.

FIG. 1 is a drawing illustrating an example of a depth data generatingsystem arranged in accordance with at least some embodiments describedherein. A depth data generating system 10 illustrated in FIG. 1 mayinclude a projection unit 11, an imaging unit 12, and a depth datagenerating apparatus 13. For the sake of convenience of explanation,three orthogonal directions in space may be represented by the x, y, andz axes of a coordinate system, with the z axis corresponding to thedepth direction of the projection unit 11 and the imaging unit 12. Theprojection unit 11 may be configured to emit a spatial light pattern.The projection unit 11 may be a conventional projector capable ofprojecting a high-definition image generated by a film medium or thelike. The projection unit 11 may alternatively be a digital projectoremploying liquid-crystal-display panels or a DLP (digital lightprocessing) projector employing Digital Micro-Mirror Devices. In such acase, display data of the spatial light pattern to be projected may beinput into the projection unit 11. The imaging unit 12 may be configuredto capture an image of one or more objects 16 through 18 on which thespatial light pattern is projected. The imaging unit 12 may be a camerathat employs an imaging device such as a CCD (Charge Coupled Device)image sensor or CMOS (Complementary Metal Oxide Semiconductor) imagesensor. The optical system parameters of the projection unit 11 and theimaging unit 12 may be adjusted such that blurring of the spatial lightpattern in the captured image monotonously increases or decreases in thedepth direction z. How to achieve such optical system parameters may bedescribed later in detail.

The imaging unit 12 may operate as a still camera to produce a singleimage at a time, or may operate as a video or motion camera to produce aseries of images continuously during an activated state. Images producedor generated by the imaging unit 12 are supplied to the depth datagenerating apparatus 13. The supply of the images may be via atransitory medium 14 or a non-transitory medium 15. The transitorymedium 14 may be a communication line through which electrical signalsare transmitted, or may be space through which electromagnetic signalsare transmitted. The non-transitory medium 15 may be a tangiblerecording medium such as a Compact Disc (CD), a Digital Versatile Disk(DVD), a memory card, or the like.

The depth data generating apparatus 13 may be configured to generatedepth data indicative of a position z in the depth direction at a pixelcorresponding to any given x, y position in an image captured by theimaging unit 12. Such depth data may be extracted from an image capturedby the imaging unit 12. Namely, the depth data generating apparatus 13may generate depth data indicative of the distance of the object 16, forexample, from the imaging unit 12 by analyzing the image captured by theimaging unit 12. The depth data generating apparatus 13 may generate adepth map indicative of depths at all the pixels in the image.

In order to generate depth data, the projection unit 11 may calculate avalue of a pre-determined spatial frequency component of the image in alocal image area around a pixel of interest. The projection unit 11 maythen determine depth data corresponding to the calculated value of thespatial frequency component by utilizing a preset relationship betweendepths and values of the spatial frequency component. As previouslynoted, the blurring of the spatial light pattern in the captured imagemay monotonously increase or decrease in the depth direction z. In sucha case, the value of the predetermined spatial frequency component maymonotonously increase or decrease in the depth direction z. Therelationship between depths and values of the predetermined spatialfrequency component may be provided as a mathematical formula.Alternatively, the relationship between depths and values of thepredetermined spatial frequency component may be obtained in advance andstored as a lookup table or the like in memory. This lookup table may bereferred to when determining depth data, so that a depth valuecorresponding to the value of the predetermined spatial frequencycomponent calculated for a pixel of interest may be located in thetable.

FIG. 2 is a drawing illustrating an example of the spatial light patternarranged in accordance with at least some embodiments described herein.In this example, the spatial light pattern may be a checkerboard patternin which black square areas and white square areas are arrangedalternately both in the vertical direction and in the horizontaldirection. The projection unit 11 and the imaging unit 12 may beconfigured to have substantially the same angle of view. In such a case,each square area of the spatial light pattern projected by theprojection unit 11 on a surface may appear to have a constanttwo-dimensional size in an image taken by the imaging unit 12 regardlessof the distance z of the surface from the projection unit 11. That is,in the example illustrated in FIG. 1, the spatial light patternprojected on the surface of the object 16 may appear to have the samesize as the spatial light pattern projected on the surface of the object17 or 18 when viewed by the imaging unit 12. The projection unit 11 maybe configured to have a focus position at a point z_(pf) in the depthdirection such that the focus point z_(pf) is nearer than any objects(e.g., 16, 17, 18) that may be captured by the imaging unit 12. Thisarrangement may ensure that the blurring of the spatial light patternmonotonously increases in the depth direction z on an object surfacesituated at depth z. The projection unit 11 may have a focus positionthat is farther away from the projection unit 11 than any objects. Thisarrangement may ensure that the blurring of the spatial light patternmonotonously decreases in the depth direction z on an object surfacesituated at depth z.

FIG. 3 is a drawing illustrating an example of the blurring of a spatiallight pattern that monotonously increases in the depth direction z on anobject surface arranged in accordance with at least some embodimentsdescribed herein. In FIG. 3, an image 30 captured by the imaging unit 12may include an image 31 of the object 16 (see FIG. 1), an image 32 ofthe object 17, and an image 33 of the object 18. As illustrated in FIG.1, the object 16, the object 17, and the object 18 may be arranged inthe depicted order in the depth direction z, with the object 16 beingthe nearest to the imaging unit 12. In such a case, a spatial lightpattern 34 projected on the nearest object 16 may be the sharpest, and aspatial light pattern 36 projected on the farthest object 18 may be themost blurred, with a spatial light pattern 35 projected on the object 17being in the middle in terms of sharpness. Depth data may be extractedfrom the image 30 by analyzing, in a local image area of interest, thevalue of the predetermined frequency component which may be a functionof the degree of blurring of the spatial light patterns 34, 35, and 36appearing in the image 30.

The spatial light pattern projected by the projection unit 11 may beconfigured such that the spatial light patterns 34, 35, and 36 appearingin the captured image 30 are so fine that a person looking at the image30 may not recognize the existence of the spatial light patterns. Such afine light pattern may be produced by a digital projector which employsa liquid-crystal-display panel having a large number of pixels such as1280×1024 pixels, 1920×1080 pixels, or the like. Under normal viewingconditions, human view may not perceive details as small as the size ofa few pixels. That is, the spatial light patterns appearing in the imagemay be periodic at spatial frequency that is higher than spatialfrequencies perceivable to human vision when the image is viewed. Thisarrangement may ensure that the quality of the image 30 from anaesthetic point of view be not undermined by the superimposed spatiallight patterns. It may be noted, however, that the size of each squarearea of the checkerboard pattern may be at least twice as large as thepixel size both in the x direction and in the y direction. Thisarrangement may ensure that a proper sampling pitch be achieved by theimaging unit 12 in order to extract accurate information about thefrequency components of the spatial light pattern.

The description provided above in respect of the spatial light patternappearing in the image 30 of FIG. 3 has not taken into account thefocusing of the imaging unit 12. If the imaging unit 12 is a pinholecamera, for example, the imaging unit 12 may be able to focus on anyobject at any distance. In such a case, a spatial light patternprojected on a given object surface may appear blurred in the capturedimage as it is on the object surface. That is, the focusing of theimaging unit 12 may be disregarded under some circumstances.

In reality, however, the characteristics of the optical system of theimaging unit 12 may affect the blurring of the spatial light patternthat appears in a captured image. That is, the focusing of the imagingunit 12 in addition to the focusing of the projection unit 11 may needto be taken into account when generating depth data from the imagecaptured by the imaging unit 12. In some circumstances, the imaging unit12 may focus on the object 16 that is an object of interest. In othercircumstances, the imaging unit 12 may focus on the object 17 that is anobject of interest, and may then focus on the object 18 that is a nextobject of interest. How to take into account the focusing of the imagingunit 12 and changes in its focus position are further described below.It may be noted here, however, that despite the focusing of the imagingunit 12, provision may easily be made such that the blurring of thespatial light pattern in the captured image monotonously increases ordecreases in the depth direction z. Such monotonous characteristics maybe provided by ensuring that a monotonous blurring increase (ordecrease) in the depth direction caused by the projection unit 11 bepredominant over a blurring change in the depth direction caused by theimaging unit 12.

FIG. 4 is a block diagram illustrating an example of the configurationof the depth data generating apparatus 13 arranged in accordance with atleast some embodiments described herein. The depth data generatingapparatus 13 illustrated in FIG. 4 may include an image receiving unit41, a frequency component calculating unit 42, a reflectance correctingunit 43, a depth map generating unit 44, and a frequency detecting unit45.

FIG. 5 is a flowchart illustrating a method for generating depth dataperformed by the depth data generating apparatus 13 arranged inaccordance with at least some embodiments described herein. As depicted,an example method may include one or more operations, actions, orfunctions as illustrated by one or more of blocks S1, S2, S3, S4, S5and/or S6. As the method in FIG. 5 could be implemented using, forexample, the depth data generating apparatus 13 depicted in FIG. 4, themethod for generating depth data will be described by referring to FIG.4 and FIG. 5.

The image receiving unit 41 may receive from the imaging unit 12 animage of one or more objects on which a spatial light pattern isprojected. In this image, the blurring of the spatial light pattern maymonotonously increase or decrease in the depth direction. Processing maybegin at block S1. At block S1, the frequency detecting unit 45 maydivide the image into blocks, each of which may be a rectangular imagearea comprised of an array of pixels. Processing may continue from blockS1 to block S2.

At block S2, the frequency detecting unit 45 may apply orthogonaltransformation, such as a Fourier transform or a Hadamard transform, toeach block to obtain frequency component F(u,v) for each block. Here,F(u,v) may be a discrete Fourier transform (or Hadamard transform, orany other orthogonal transform) of pixel data b(x,y), which representsthe pixel values of the pixel array corresponding to a block ofinterest. Coordinates x and y may represent pixel positions within theblock of interest. Variable u represents spatial frequency in the xdirection, and variable v represents spatial frequency in the ydirection. Processing may continue from block S2 to block S3.

At block S3, the frequency detecting unit 45 may add up F(u,v) of allthe blocks to obtain S(u,v), and may find the largest value of S(u,v) ina frequency range above a threshold frequency. S(u,v) may be the sum ofF(u,v) of the first block, F(u,v) of the second block, F(u,v) of thethird block, . . . , and F(u,v) of the N-th block when there are Nblocks in the image. Since the spatial light pattern periodic at certainspatial frequency is included in the image, S(u,v) may have a localmaximum value at this spatial frequency within a frequency range above acertain threshold frequency. This threshold frequency may be determinedby taking into account the spatial frequency at which the spatial lightpattern is periodic. For example, the threshold frequency may be setlower than the above-noted special frequency but higher than most of thefrequency spectrum of image information attributable to the capturedscene. Since most of the frequency spectrum of image informationattributable to the captured scene (i.e., excluding image informationattributable to the projected spatial light pattern) resides at lowerfrequencies, the provision of such a threshold frequency may make itpossible to detect the local maximum of the spectrum attributable to theprojected spatial light pattern. In this manner, spatial frequency(u_(m), v_(m)) at which the spatial light pattern is periodic may bedetected. Processing may continue from block S3 to block S4.

At block S4, the frequency component calculating unit 42 may applyorthogonal transformation, such as the Fourier transform or the Hadamardtransform, to a local image area around each pixel of interest atcoordinates (x,y) to obtain frequency component F_(x,y)(u_(m),v_(m)).This local image area may be a rectangular image area centered atcoordinates (x,y). Here, F_(x,y)(u,v) may be a discrete Fouriertransform (or Hadamard transform, or any other orthogonal transform) ofpixel data, which are the pixel values of the pixel array of the localimage area. Processing may continue from block S4 to block S5.

At block S5, the reflectance correcting unit 43 may correctF_(x,y)(u_(m),v_(m)) for different object reflectance. The spatial lightpattern projected by the projection unit 11 may be reflected by anobject surface to reach the imaging unit 12. The intensity of thereflected light may depend on the reflectance of the object surface. Asa result, pixel values (i.e., luminance values) of the captured imageare dependent on the reflectance of each object surface. As thereflectance of the object surface increases, the pixel value of a pixelsituated within an image area corresponding to this object surface mayalso increase. It follows that the value of F_(x,y)(u_(m), v_(m)) mayalso depend on object surface reflectance. The reflectance correctingunit 43 may remove variation in F_(x,y)(u_(m),v_(m)) caused bydifferences in object surface reflectance. This correction process maybe described later in detail. Processing may continue from block S5 toblock S6.

At block S6, the depth map generating unit 44 may generate a depth mapbased on the corrected F_(x,y)(u_(m),v_(m)) and a lookup table. Thelookup table may indicate a relationship between depths (i.e, positionsin the z direction) and the values of the spatial frequency componentF_(x,y)(u_(m), v_(m)). This lookup table may be prepared in advance andstored in memory.

FIG. 6 is a block diagram illustrating an example of the configurationof the reflectance correcting unit 43 arranged in accordance with atleast some embodiments described herein. The reflectance correcting unit43 illustrated in FIG. 6 may include a global average luminancecalculating unit 61, a correcting unit 62, and a local average luminancecalculating unit 63.

FIG. 7 is a flowchart illustrating a method for performing luminancecorrection performed by the reflectance correcting unit 43 arranged inaccordance with at least some embodiments described herein. As depicted,an example method may include one or more operations, actions, orfunctions as illustrated by one or more of blocks S21, S22 and/or S23.As the method in FIG. 7 could be implemented using, for example, thereflectance correcting unit 43 depicted in FIG. 6, the method forperforming luminance correction will be described by referring to FIG. 6and FIG. 7. Processing may begin at block S21.

At block S21, the global average luminance calculating unit 61 maycalculate average luminance DC_(A) of the entire image. The averageluminance DC_(A) of the entire image may be calculated by adding up allthe pixel values and by dividing the obtained sum by the number ofpixels. Alternatively, the average luminance DC_(A) of the entire imagemay be calculated by using a direct current component F_(x,y)(0,0) thatmay be obtained together with F_(x,y)(u_(m),v_(m)). Since the directcurrent component F_(x,y)(0,0) is proportional to the sum of pixelvalues within the local image area, the global average luminancecalculating unit 61 may use F_(x,y)(0,0) to obtain the average luminanceDC_(A) of the entire image. Processing may continue from block S21 toblock S22.

At block S22, the local average luminance calculating unit 63 maycalculate average luminance DC_(xy) of a local image area around eachpixel of interest situated at coordinates (x,y). The average luminanceDC_(xy) of the local image area may be calculated by adding up all thepixel values of the local image area and by dividing the obtained sum bythe number of pixels. Alternatively, the average luminance DC_(xy) ofthe local image area may be directly obtained from the direct currentcomponent F_(x,y)(0,0). Processing may continue from block S22 to blockS23.

At block S23, the correcting unit 62 may multiply F_(x,y)(u_(m),v_(m))by DC_(A)/DC_(xy) for luminance correction. It may be noted that localaverage luminance DC_(xy) of a local image area may be proportional tothe reflectance of an object surface corresponding to this local imagearea. Further, the value of F_(x,y)(u_(m),v_(m)) may also beproportional to the reflectance of the object surface corresponding tothis local image area. Accordingly, multiplication by DC_(A)/DC_(xy) mayremove the effect of object-surface reflectance variation fromF_(x,y)(u_(m),v_(m)). It may also be noted that for the purpose ofremoving the effect of object-surface reflectance variation, it maysuffice to multiply F_(x,y)(u_(m), v_(m)) by 1/DC,_(xy).

FIG. 8 is a block diagram illustrating an example of the configurationof the depth map generating unit 44 arranged in accordance with at leastsome embodiments described herein. The depth map generating unit 44illustrated in FIG. 8 includes a camera parameter obtaining unit 81, anA(z) calculating unit 82, an LUT generating unit 83, a depth calculatingunit 84, and an LUT (lookup table) 85 indicative of B(z). A(z) mayrepresent a relationship between position in the depth direction and theblurring of the spatial light pattern caused by the projection unit 11that projects the spatial light pattern. B(z) may represent arelationship between position in the depth direction and the blurring ofthe spatial light pattern caused by the imaging unit 12 that captures animage.

FIG. 9 is a flowchart illustrating a method for determining depth dataperformed by the depth map generating unit 44 arranged in accordancewith at least some embodiments described herein. As depicted, an examplemethod may include one or more operations, actions, or functions asillustrated by one or more of blocks S31, S32, S33, S34 and/or S35. Thedepicted example method determines depth data by combining the effect offocusing of the projection unit 11 with the effect of focusing of theimaging unit 12. As the method in FIG. 9 could be implemented using, forexample, the depth map generating unit 44 depicted in FIG. 8, the methodfor determining depth data will be described by referring to FIG. 8 andFIG. 9. Processing may begin at block S31.

At block S31, the camera parameter obtaining unit 81 may obtain a focusposition z_(cf), in the z direction and an aperture size D from theimaging unit 12. The camera parameter obtaining unit 81 may also obtaina focus position (x_(cf),y_(cf)), at which point in the x-y plane theimaging unit 12 focuses on an object surface. The aperture size D maytypically be provided in the form of a focal length f and an f-number N.The aperture size (i.e., the diameter of an entrance pupil) D may thenbe calculated as D=f/N. Processing may continue from block S31 to blockS32.

At block S32, the A(z) calculating unit 82 may calculate adepth-dependent amplitude variation A(z) of the spatial patternfrequency caused by the camera optical system (i.e., the imaging unit12) as a function of depth z based on the focus position z _(cf) and theaperture size D. This depth-dependent amplitude variation A(z) mayrepresent the amplitude of the spatial frequency (u_(m),v_(m)) of thespatial light pattern on a surface positioned at depth z as viewed bythe imaging unit 12 focusing on the focus position z_(cf). This spatiallight pattern may be an ideal spatial light pattern having no blurringon the surface positioned at depth z. Namely, the depth-dependentamplitude variation A(z) may take into account only the blurring of thespatial light pattern caused by the imaging unit 12 under the conditionin which the blurring of the spatial light pattern caused by theprojection unit 11 is nonexistent.

The depth-dependent amplitude variation A(z) may be obtained as follows.A point-spread function of the imaging unit 12 focusing on the distancez_(cf) may be represented as psf(x,y,z) that assumes a value of(2(z_(cf)+z)/Dz)²/pi (pi=3.14 . . . ) within a circular area having adiameter of Dz/(z_(cf)+z) and that assumes zero outside this circulararea. The spatial light pattern positioned at depth z may then appear inan image captured by the imaging unit 12 as p(x,y)=p₀(x,y)*psf(x,y,z)where p₀(x,y) is the spatial light pattern at position z_(cf) and thesymbol “*” represents a convolution operation. An orthogonal transform(e.g., Fourier transform or Hadamard transform) of p(x,y) with respectto x and y may be represented as P(u,v)=P₀(u,v)PSF(u,v,z) where P₀(u,v)and PSF(u,v,z) are orthogonal transforms of p₀(x,y) and psf(x,y,z),respectively. Then, the depth-dependent amplitude variation A(z) may beequal to P(u_(m),v_(m)) where (u_(m),v_(m)) is the spatial frequency atwhich the spatial light pattern is periodic. Processing may continuefrom block S32 to block S33.

At block S33, the LUT generating unit 83 may create a lookup table Bc(z)indicative of a relationship between F_(x,y)(u_(m),v_(m)) and depth zbased on A(z) and B(z) where B(z) represents the blurring of the spatiallight pattern caused by the projection unit 11. B(z) may be obtained inadvance as follows. First, the projection unit 11 may project thespatial light pattern (e.g., checkerboard pattern) on a blank, flatsurface (e.g., white, flat surface) positioned at depth z while focusingon the depth z_(pf) as described in connection with FIG. 1. An image ofthis spatial light pattern on the surface positioned at depth z may thenbe captured by the imaging unit 12 focusing on this surface positionedat depth z. A spatial light pattern p(x,y,z) in the captured image mayinclude blurring caused by the projection unit 11, but may not includeblurring caused by the imaging unit 12. When an orthogonal transform ofp(x,y,z) with respect to x and y is denoted as P(u,v,z), B(z) may berepresented as P(u_(m),v_(m),z). B(z) obtained in this manner in advancemay be stored as a lookup table B(z) in memory. The depth-dependentamplitude variation that factors in both the blurring caused by theprojection unit 11 and the blurring caused by the imaging unit 12 maythen be represented as A(z)B(z). Namely, A(z)B(z) may be stored as alookup table Bc(z). It may be noted that B(z) may alternatively beobtained as the depth-dependent characteristics of a point-spreadfunction by projecting a single light spot rather than the spatial lightpattern such as a checkerboard pattern. It may further be noted thatB(z) may alternatively be obtained as a mathematical formula thatapproximates a point-spread function of the projection unit 11. Thepoint spread function may be similar to psf(x,y,z) previously described,or may be represented by a Gaussian function. When B(z) is provided as amathematical formula, Bc(z) may also be obtained as a mathematicalformula rather than as a lookup table. Processing may continue fromblock S33 to block S34.

At block S34, the depth calculating unit 84 may standardizeF_(x,y)(u_(m), v_(m)) for a luminance factor. As previously described,the value of the frequency component as corrected by the reflectancecorrecting unit 43 for reflectance variation may be F_(x,y)(u_(m),v_(m))DC_(A)/DC_(xy)),. The focus position (x_(cf),y _(cf)) at whichpoint in the x-y plane the imaging unit 12 focuses on an object surfacemay be obtained from the imaging unit 12 in block S1 of FIG. 9. Thevalue of F_(x,y)(u_(m),v_(m))DC_(A)/DC, at this focus position(x_(cf),y_(cf)) may be denoted as Rxy. Then, the depth calculating unit84 may standardize the value of F_(x,y)(u _(m), v_(m))DC_(A)/DC_(xy) atany given spatial coordinates (x,y) in the captured image by multiplyingthis value by Bc(z_(cf))/Rxy. This standardization process may ensurethat the amplitude of the spatial frequency (u_(m),v_(m)) at the focusposition (x,_(cf),y_(cf)) at the focus depth z_(cf) be equal toBc(z_(cf)). Processing may continue from block S34 to block S35.

At block S35, the depth calculating unit 84 may determine a depth ateach pixel position based on the standardized F_(x,y)(u_(m),v_(m)) andBc(z). Namely, the depth calculating unit 84 may refer to the lookuptable Bc(z) to find a depth z associated with the (F_(x,y)(u_(m),v_(m))DC_(A)/DC_(xy))x(Bc(z_(cf))/Rxy). The depth z found in the lookuptable Bc(z) may be the depth at which an object surface corresponding tothe pixel position (x,y) is positioned in the scene captured by theimaging unit 12.

In the manner described above, the blurring of the spatial light patterncaused by the projection unit 11 and the blurring of the spatial lightpattern caused by the imaging unit 12 may both be taken into account ina depth map generated by the depth map generating unit 44. Further, aspreviously noted, despite the focusing of the imaging unit 12, provisionmay easily be made such that the blurring of the spatial light patternin the captured image monotonously increases or decreases in the depthdirection z. Such monotonous characteristics may be provided by ensuringthat a monotonous blurring increase (or decrease) in the depth directioncaused by the projection unit 11 be predominant over a blurring changein the depth direction caused by the imaging unit 12.

FIG. 10 is a drawing illustrating an example of a monotonous blurringincrease in the depth direction arranged in accordance with at leastsome embodiments described herein. The horizontal axis may representdepth z, and the vertical axis may represent the amplitude of thespatial frequency (u_(m),v_(m)). A characteristic curve 91 maycorrespond to A(z), which may represent the blurring of the spatiallight pattern caused by the imaging unit 12. The imaging unit 12 mayhave a focus position at an object of interest, so that thecharacteristics curve 91 may have a maximum peak at this focus positionin the depth direction z. A characteristic curve 92 may correspond toB(z), which may represent the blurring of the spatial light patterncaused by the projection unit 11. The projection unit 11 may have afocus position at the nearest point in the depth direction z, so thatthe characteristics curve 92 monotonously falls as depth z increases.Namely, the blurring represented by the characteristic curve 92 maymonotonously increase as depth z increases.

As illustrated in FIG. 10, provision may be made such that a monotonousblurring increase in the depth direction caused by the projection unit11 be predominant over a blurring change in the depth direction causedby the imaging unit 12. Accordingly, a characteristics curve 93 whichcorresponds to Bc(z)=A(z)B(z) may monotonously fall as depth zincreases. That is, blurring obtained by combining the blurring causedby the projection unit 11 and the blurring caused by the imaging unit 12may monotonously increase as depth z increases. With such blurringcharacteristics, depth data may be uniquely determined from a givenamplitude of the spatial frequency (u_(m),v_(m)).

References made in this disclosure to the term “responsive to” or “inresponse to” are not limited to responsiveness to only a particularfeature and/or structure. A feature may also be responsive to anotherfeature and/or structure and also be located within that feature and/orstructure. Moreover, when terms or phrases such as “coupled” or“responsive” or “in response to” or “in communication with”, etc. areused herein or in the claims that follow, these terms should beinterpreted broadly. For example, the phrase “coupled to” may refer tobeing communicatively, electrically and/or operatively coupled asappropriate for the context in which the phrase is used.

Some portions of the foregoing detailed description are presented interms of algorithms or symbolic representations of operations on databits or binary digital signals stored within a computing system memory,such as a computer memory. These algorithmic descriptions orrepresentations are examples of techniques used by those of ordinaryskill in the data processing arts to convey the substance of their workto others skilled in the art. An algorithm is here, and generally, isconsidered to be a self-consistent sequence of operations or similarprocessing leading to a desired result. In this context, operations orprocessing involve physical manipulation of physical quantities.Typically, although not necessarily, such quantities may take the formof electrical or magnetic signals capable of being stored, transferred,combined, compared or otherwise manipulated. It has proven convenient attimes, principally for reasons of common usage, to refer to such signalsas bits, data, values, elements, symbols, characters, terms, numbers,numerals or the like. It should be understood, however, that all ofthese and similar terms are to be associated with appropriate physicalquantities and are merely convenient labels. Unless specifically statedotherwise, as apparent from the following discussion, it is appreciatedthat throughout this specification discussions utilizing terms such as“processing,” “computing,” “calculating,” “determining” or the likerefer to actions or processes of a computing device, that manipulates ortransforms data represented as physical electronic or magneticquantities within memories, registers, or other information storagedevices, transmission devices, or display devices of the computingdevice.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In one embodiment,several portions of the subject matter described herein may beimplemented via Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), digital signal processors (DSPs), orother integrated formats. However, those skilled in the art willrecognize that some aspects of the embodiments disclosed herein, inwhole or in part, can be equivalently implemented in integratedcircuits, as one or more computer programs running on one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs running on one or more processors(e.g., as one or more programs running on one or more microprocessors),as firmware, or as virtually any combination thereof, and that designingthe circuitry and/or writing the code for the software and or firmwarewould be well within the skill of one of skill in the art in light ofthis disclosure. In addition, those skilled in the art will appreciatethat the mechanisms of the subject matter described herein are capableof being distributed as a program product in a variety of forms, andthat an illustrative embodiment of the subject matter described hereinapplies regardless of the particular type of signal bearing medium usedto actually carry out the distribution. Examples of a signal bearingmedium include, but are not limited to, the following: a recordable typemedium such as a flexible disk, a hard disk drive (HDD), a Compact Disc(CD), a Digital Versatile Disk (DVD), a digital tape, a computer memory,etc.; and a transmission type medium such as a digital and/or an analogcommunication medium (e.g., a fiber optic cable, a waveguide, a wiredcommunications link, a wireless communication link, etc.).

Those skilled in the art will recognize that it is common within the artto describe devices and/or processes in the fashion set forth herein,and thereafter use engineering practices to integrate such describeddevices and/or processes into data processing systems. That is, at leasta portion of the devices and/or processes described herein can beintegrated into a data processing system via a reasonable amount ofexperimentation. Those having skill in the art will recognize that atypical data processing system generally may include one or more of asystem unit housing, a video display device, a memory such as volatileand non-volatile memory, processors such as micro-processors and digitalsignal processors, computational entities such as operating systems,drivers, graphical user interfaces, and applications programs, one ormore interaction devices, such as a touch pad or screen, and/or controlsystems including feedback loops and control motors (e.g., feedback forsensing position and/or velocity; control motors for moving and/oradjusting components and/or quantities). A typical data processingsystem may be implemented utilizing any suitable commercially availablecomponents, such as those typically found in datacomputing/communication and/or network computing/communication systems.

The herein described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely exemplary, and that in fact many other architectures can beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected”, or“operably coupled”, to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable”, to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

It will be understood by those within the art that, in general, termsused herein, and especially in the appended claims (e.g., bodies of theappended claims) are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims may contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations. However, the use of such phrases should not be construed toimply that the introduction of a claim recitation by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim recitation to implementations containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should typically be interpreted to mean “atleast one” or “one or more”); the same holds true for the use ofdefinite articles used to introduce claim recitations. In addition, evenif a specific number of an introduced claim recitation is explicitlyrecited, those skilled in the art will recognize that such recitationshould typically be interpreted to mean at least the recited number(e.g., the bare recitation of “two recitations,” without othermodifiers, typically means at least two recitations, or two or morerecitations). Furthermore, in those instances where a conventionanalogous to “at least one of A, B, and C, etc.” is used, in generalsuch a construction is intended in the sense one having skill in the artwould understand the convention (e.g., “a system having at least one ofA, B, and C” would include but not be limited to systems that have Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). In those instances where aconvention analogous to “at least one of A, B, or C, etc.” is used, ingeneral such a construction is intended in the sense one having skill inthe art would understand the convention (e.g., “a system having at leastone of A, B, or C” would include but not be limited to systems that haveA alone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). It will be furtherunderstood by those within the art that virtually any disjunctive wordand/or phrase presenting two or more alternative terms, whether in thedescription, claims, or drawings, should be understood to contemplatethe possibilities of including one of the terms, either of the terms, orboth terms. For example, the phrase “A or B” will be understood toinclude the possibilities of “A” or “B” or “A and B.”

It should also be understood that, the term “optimize” may includemaximization and/or minimization. The term “minimization” and/or thelike as used herein may include a global minimum, a local minimum, anapproximate global minimum, and/or an approximate local minimum.Likewise, it should also be understood that, the term “maximization”and/or the like as used herein may include an global maximum, a localmaximum, an approximate global maximum, and/or an approximate localmaximum.

Reference in the specification to “an implementation,” “oneimplementation,” “some implementations,” or “other implementations” maymean that a particular feature, structure, or characteristic describedin connection with one or more implementations may be included in atleast some implementations, but not necessarily in all implementations.The various appearances of “an implementation,” “one implementation,” or“some implementations” in the preceding description are not necessarilyall referring to the same implementations.

While certain example techniques have been described and shown hereinusing various methods and systems, it should be understood by thoseskilled in the art that various other modifications may be made, andequivalents may be substituted, without departing from claimed subjectmatter. Additionally, many modifications may be made to adapt aparticular situation to the teachings of claimed subject matter withoutdeparting from the central concept described herein. Therefore, it isintended that claimed subject matter not be limited to the particularexamples disclosed, but that such claimed subject matter also mayinclude all implementations falling within the scope of the appendedclaims, and equivalents thereof.

1. A method of generating depth data, comprising: calculating a value ofa spatial frequency component of image data in a local image area arounda pixel of interest, wherein the image data is representative of one ormore objects on which a spatial light pattern is projected and blurringof the spatial light pattern in the image data increases in a firstdepth direction and decreases in a second depth direction; anddetermining depth data corresponding to the calculated value of thespatial frequency component by utilizing a preset relationship betweendepths and values of the spatial frequency component.
 2. The method ofclaim 1, further comprising detecting a spatial frequency at which aspatial frequency spectrum of the image has a maximum value in afrequency range above a threshold frequency, wherein the calculatedvalue of the spatial frequency component is a value of a spatialfrequency component at the detected spatial frequency, and wherein thespatial light pattern in the image data is at least periodic at aspatial frequency above the threshold frequency.
 3. The method of claim1, wherein the spatial frequency at which the spatial light pattern inthe image is periodic is higher than spatial frequencies that areperceivable to human vision when the image data is viewed.
 4. The methodof claim 1, wherein the local image area is a rectangular area, and thecalculating a value of a spatial frequency component comprisescalculating spatial frequency components by applying orthogonaltransformation to the rectangular area.
 5. The method of claim 1,further comprising correcting the calculated value of the spatialfrequency component for a reflectance of an object appearing in thelocal image area of the image.
 6. The method of claim 5, wherein thecorrecting the calculated value comprises dividing the calculated valueby an average luminance of the image in the local image area.
 7. Themethod of claim 1, further comprising: calculating the presetrelationship between depths and values of the spatial frequencycomponent based on first data and second data, wherein the first data isindicative of a first relationship between position in the depthdirection and blurring of the spatial light pattern caused by aprojection system for projecting the spatial light pattern and thesecond data is indicative of a second relationship between position inthe depth direction and blurring of the spatial light pattern caused byan imaging system for capturing the image.
 8. The method of claim 7,wherein the second data is determined by: calculating the blurring ofthe spatial light pattern caused by the imaging system with respect todifferent positions in the depth direction based on data indicative of afocus position of the imaging system in the depth direction and a lensaperture size of the imaging system.
 9. The method of claim 7, whereinthe first data is determined by: projecting the spatial light patternfrom the projection system to a flat, monotonous surface; capturing animage of the spatial light pattern projected on the surface by use ofthe imaging system focusing on the surface; and performing theprojecting the spatial light pattern and the capturing the image of thespatial light patter with respect to flat, monotonous surfaces placed atdifferent positions in the depth direction.
 10. An apparatus forgenerating depth data, comprising: a frequency component calculatingunit configured to calculate a value of a spatial frequency component ofimage data in a local image area around a pixel of interest, wherein theimage data is representative of one or more objects on which a spatiallight pattern is projected and blurring of the spatial light pattern inthe image data increases in a first depth direction and decreases in asecond depth direction; and a depth map generating unit configured todetermine depth data corresponding to the calculated value of thespatial frequency component by utilizing a preset relationship betweendepths and values of the spatial frequency component.
 11. The apparatusof claim 10, further comprising a frequency detecting unit configured todetect a spatial frequency at which a spatial frequency spectrum of theimage has a maximum value in a frequency range above a thresholdfrequency, wherein the calculated value of the spatial frequencycomponent is a value of a spatial frequency component at the detectedspatial frequency, and wherein the spatial light pattern in the imagedata is at least periodic at a spatial frequency above the thresholdfrequency.
 12. The apparatus of claim 10, wherein the spatial frequencyat which the spatial light pattern in the image is periodic is higherthan spatial frequencies that are perceivable to human vision when theimage is viewed.
 13. The apparatus of claim 10, wherein the frequencycomponent calculating unit is configured to calculate spatial frequencycomponents by applying orthogonal transformation to a rectangular areathat is the local image area.
 14. The apparatus of claim 10, furthercomprising a reflectance correcting unit configured to correct thecalculated value of the spatial frequency component for a reflectance ofan object appearing in the local image area of the image.
 15. Theapparatus of claim 14, wherein the reflectance correcting unit isconfigured to divide the calculated value by an average luminance of theimage in the local image area.
 16. The apparatus of claim 10, whereinthe depth map generating unit comprises: a relationship calculating unitconfigured to calculate the preset relationship between depths andvalues of the spatial frequency component based on first data and seconddata, wherein the first data is indicative of a first relationshipbetween position in the depth direction and blurring of the spatiallight pattern caused by a projection system for projecting the spatiallight pattern and the second data is indicative of a second relationshipbetween position in the depth direction and blurring of the spatiallight pattern caused by an imaging system for capturing the image. 17.The apparatus of claim 16, unit configured to obtain data indicative ofa focus position of the imaging system in the depth wherein the blurringdata obtaining unit is configured to calculate the blurring of thespatial light pattern caused by the imaging system with respect todifferent positions in the depth direction based on a focus position ofthe imaging system and a lens aperture size of the imaging system.
 18. Asystem for generating depth data, comprising: a projection unitconfigured to project a spatial light pattern; and an imaging unitconfigured to capture an image of one or more objects on which thespatial light pattern is projected, wherein blurring of the spatiallight pattern in the image monotonously increases or decreases in adepth direction; such that depth data corresponding to a calculatedvalue of a spatial frequency component of the image in a local imagearea around a pixel of interest can be determined by utilizing a presetrelationship between depths and values of the spatial frequencycomponent.
 19. The system of claim 18, wherein the spatial light patterncomprises a checkerboard pattern in which a size of each square area ofthe checkerboard pattern is at least twice as large as a pixel size inan x direction and in a y direction.
 20. The system of claim 18, whereinthe captured image is operable for generating a stereoscopic image.